(1) Field of the Invention
The present invention relates to an image reading method suitable for increasing image resolution and an image expansion method for expanding and reproducing a read image.
(2) Description of the Related Art
Recently, efforts have been actively made to enhance the resolution of moving pictures and still pictures shot by a digital still camera or the like for obtaining sharper images. As one of the efforts, a method whereby a plurality of images having displacement from one another are synthesized to restore high frequency components has been studied as a method for obtaining a sharp high-resolution image close to its original image. With this method, a plurality of images, which are temporally consecutive in a moving image, can be used to enhance the resolution, and this method may be applicable for a wide range of uses, such as enhancement of the resolution of a moving image shot by a digital still camera and a video camera. Hereinafter, processing whereby a high-resolution image is generated from a plurality of low-resolution images having displacement from one another is referred to as super-resolution.
Although there are a number of super-resolution techniques, reconstruction type super-resolution is widely used whereby the pixel value of the high-resolution image is sequentially updated by repetitive processing in order to stably obtain a high-quality and high-resolution image. An example of reconstruction type super-resolution is a Maximum A Posteriori (MAP) method (see, e.g., Japanese Unexamined Patent Application Publication 2000-339450). The MAP method is a method for performing reconstruction of a high-resolution image which maximizes a posterior probability when a high-resolution image generated by a bicubic method, a nearest neighbor method, or the like is taken as an initial value, and a plurality of low-resolution images, which are observed images, is the condition. A posteriori probability is indicated by a performance function, which is made of an error term and a convergent term. When the imaging model is assumed, the error term represents the square error between a pixel value estimated from a high-resolution image based on an imaging model, and a pixel value of an positioned low-resolution image, and the convergent term represents a constraint based on advance information such as an image is assumed to be smooth at all. Here, when a low-resolution image is sufficiently band-limited, super-resolution has less effect because a high-resolution image can be restored from the low-resolution image, therefore, it is preferable that the low-resolution image to which super-resolution is applied includes many high frequency components, in other words, aliasing components. Accordingly, it is important to satisfy the following two requirements in order to apply super-resolution.
A first requirement is that positioning between low-resolution images can be made accurately, and a second requirement is that strong aliasing exists in the low-resolution image.
On the other hand, it is difficult for a common digital still camera or the like to read a high-resolution image, which is shot by an imaging device, at a high frame rate due to processing power, therefore, pixel mixture or decimation reading is used as a method for reducing the amount of data to be read. In this case, since a high-resolution image is read as a low-resolution image, a technique whereby a low-resolution image after being read is expanded to have a high image quality by super-resolution receives attention. As an example, a method whereby super-resolution is performed using both a mixed image read by pixel mixture, and a decimated image read by decimation reading is described in Nonpatent document 1 (“Multi-mixture Imaging and Its Application to Super-resolution”, PRMU2005-274 (2006-3)).
An image expansion method in the Nonpatent document 1 will be described below as a prior art example.
In a conventional image expansion method, two types of low-resolution images: a mixed image and a decimated image are read from an imaging device that can take a picture at high resolution. The mixed image is an image that is read after a plurality of pixels in a region to be read are mixed, and a decimated image is an image obtained by reading only one pixel among pixels within a given range. Here, since high frequency components are suppressed by pixel mixture in the mixed image, an effect of aliasing is suppressed. In an image where the effect of aliasing is suppressed, accurate positioning can be achieved. Meanwhile, strong aliasing is included in the decimated image. Accordingly, the mixed image can be used for positioning, and the decimated image can be used for super-resolution reconstruction to satisfy the two requirements needed for super-resolution.
FIG. 1 is a block diagram showing the configuration of an image expansion device 500 that performs a conventional image expansion method. As shown in FIG. 1, the image expansion device 500 has an image storage unit 510, an image reproducing unit 520, and a memory 530.
The image storage unit 510 is constituted by an imaging device such as Charge Coupled Devices (CCD) and Complementary Metal Oxide Semiconductor (CMOS), and has an image input unit 511, and an image reading unit 512. The image input unit 511 shoots an input image 501. The image reading unit 512 reads a decimated image 502 and a mixed image 503 from the input image 501, which is shot by the image input unit 511, and stores each image in the memory 530.
Meanwhile, the image reproducing unit 520 has a positioning unit 521 and an image expansion unit 522. The positioning unit 521 reads a plurality of mixed images 503 from the memory 530, positions the plurality of read mixed images 503 to calculate the amount of displacement among the images, and outputs the amount of displacement to the image expansion unit 522. Note that here the amount of displacement is represented by a motion vector 504. The image expansion unit 522 performs, based on the motion vector 504, which represents the amount of displacement, the super-resolution reconstruction to expand the image using a plurality of decimated images 502, which are read from the memory 530, and outputs an expanded image 505.
FIG. 2 is a flowchart showing the operation of the image storage unit 510, and FIGS. 3A and 3B are diagrams for describing a decimated image and a mixed image. FIG. 3A shows a decimated image, and FIG. B shows a mixed image.
First, as pixels in the decimated image, the image reading unit 512 reads pixel values at predetermined positions from the input image shot by the image input unit 511 (Step S0101). For example, as shown in FIG. 3A, when pixels are read at the rate of one pixel in every 4 pixels (indicated by black triangles in the figure), the decimated image is a low-resolution image, the resolution of which is ½ with respect to the input image 501. Then, the image reading unit 512 reads mixed pixels at the same positions as the reading positions of the decimated image from the input image 501 as pixels in the mixed image (Step S0102). For example, as shown in FIG. 3B, when pixel mixture is performed, and one pixel is calculated and read using nine pixels in an area 301, the center of the nine pixels is a discrete point (indicated by a black rectangle in the figure) of the mixed image, and the mixed image is a low-resolution image, the resolution of which is ½ with respect to the input image 501.
FIG. 4 is a flowchart showing the operation of the image reproducing unit 520.
First, the positioning unit 521 performs positioning using a plurality of mixed images to obtain the amount of displacement among images (Step S0201). Next, the image expansion unit 522 performs super-resolution reconstruction based on the motion vector 504, and expands the image (Step S0202).
However, in the conventional image expansion method, the reading positions of the mixed image and the decimated image are fixed. FIG. 5(a) is a diagram showing an example of pixel mixture in the conventional image expansion method. FIG. 5(a) shows an example that nine pixels are mixed, the center of the nine pixels to be mixed corresponding to the discrete point (indicated by a black rectangle in the figure) in the mixed image. When it is assumed that pictures from Pic1 to Pic4 in the figure represent four consecutive images (picture), the positions of the discrete points are equal in all the images. In this example, the mixed image is a low-resolution image, the resolution of which is ½ with respect to the input. Note that only part of the pixels in the picture is shown in the figure, and other pixels are omitted. FIG. 5(b) is a diagram showing the positions of the pixels in a coordinate of the read low-resolution image of the mixed image corresponding to the pictures from Pic1 to Pic4. It is clear that each grid shows the pixel positions of the low-resolution image, the discrete points in the mixed image always being mapped to the identical pixel positions in the pictures from Pic1 to Pic4. Note that, also in the decimated image, pixel values at fixed decimation positions were read.
As described above, in a conventional image expansion method, since the discrete points of the mixed image, and the decimation positions of the decimated image are fixed, no displacement is generated in the read mixed image and decimated image when an input image sequence is motionless. For super-resolution, when there is no displacement among low-resolution images, the effect is significantly lowered. As a result, in the conventional image expansion method, there is a problem that the effect of the super-resolution is significantly lowered when an input image sequence is motionless, in other words, when a subject is motionless.